Multi-touch system and method for emulating modifier keys via fingertip chords

ABSTRACT

A multi-touch system is disclosed that recognizes simultaneous touchdown of four fingers on, above, or below the home row of keys as a modifier chord and applies modifiers such as Shift, Ctrl, or Alt to subsequent touch activity until none of the chord fingertips remain touching. Touches by the thumb of the modifier chord hand that occur before any modifiable typing or clicking activity cause the modifier chord to be canceled and reinterpreted as hand resting. The Shift modifier may be released temporarily during thumb keypresses that are intermixed with typing of capitalized characters. Distributing the modifier chord touches across different zones or key rows selects multiple modifiers. In an alternative embodiment, different modifiers can be selected with different arrangements of the fingers relative to one another within the chord, irrespective of absolute hand alignment with the touch surface.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention pertains to multi-touch input systems and methods,and more particularly to a mixture of chord keying, gesture recognitionand touch typing techniques.

2. The Related Art

The primary use of chords, or simultaneous finger presses, within thedata entry art has been in chord keying schemes that map each letter ofthe alphabet or even shorthand word parts to a different fingercombination. This allows chord keyboards to have a reduced number ofkeys, often limited to a home row of keys. This in turn reduces fingertravel and potentially speeds typing. Some schemes, like U.S. Pat. No.5,281,966 to Walsh, adopt a mapping that is sensibly organized so as tobe easy to learn and remember, while others, such as U.S. Pat. No.5,642,108 to Gopher et al., emphasize long-term keying performance byassigning the most frequently entered letters of the alphabet to thosefinger combinations that are quickest and easiest to perform. In U.S.Pat. No. 5,808,567, McClound discloses a scheme for communicating withthree-finger chords. In this system, a touch of the index finger on oneof the nine regions of a small selector pad can be modified by thumband/or middle finger presses on switch pads adjacent to the selectorpad.

The recent development of multiple-touch sensitive surfaces that lackthe restrictions of distinct mechanical keys warrants a reexamination ofchording schemes. Direct adaptation of the chord keying schemes citedabove to a multi-touch surface certainly seems feasible, but may not bedesirable. U.S. Pat. No. 5,825,352 to Bisset et al. describes a touchpadwith row and column electrodes that produces pointing in response tosingle finger motion and dragging in response to two finger motion. U.S.Pat. No. 6,107,997, Ure utilizes the touch sensor array of U.S. Pat. No.5,194,862 and interprets single finger motions as pointing whileinterpreting various placements of a 2-finger chord on a grid as keyentry. In U.S. application Ser. No. 09/236,513, however, Westerman andElias take yet another approach, interpreting asynchronous touches on amulti-touch surface (MTS) as conventional single-finger typing whileinterpreting motions initiated by chords as pointing, clicking, andother gesture commands. We prefer this approach for the followingreasons: learning a few new chords for graphical manipulation is mucheasier than learning a slew of new chords for typing the whole alphabet,and graphical manipulation seems a better use of chords in today'sgraphics intensive computing environment. In dictation situations wheregreater text entry speeds are needed than can be achieved withnon-chordic keying, adopting a continuous speech recognition system fortext entry is becoming more practical than learning a chord keyingtechnique.

Non-chordic touch typing on surfaces that provide limited tactilefeedback presents its own difficulties. If the typist is not careful,the hands or individual fingers tend to drift out of alignment with thekey layout, or more particularly with the home row of keys where handsnormally rest. Reaching for punctuation and modifier keys located on theperiphery of QWERTY computer keyboard layouts exacerbates this drift.Though the Shift modifier key is not particularly far from the home rowkeys, the direction of pinky motion needed to reach Shift strongly pullsthe other fingertips off their alignment with home row. Since the Shiftmodifier key must be reached so frequently to capitalize words, eventypists using mechanical keyboards have long complained about theawkward pinky twist and ulnar deviation at the wrist necessary to holdit down. Accurately, hitting the Shift keys becomes, if anything, moreawkward on a relatively smooth surface that does not give like amechanical key.

In the related ergonomic and chord keyboard art exemplified by FIG. 2modifier keys such as Shift, Ctrl, and Alt are often allocated to thethumbs (e.g. U.S. Pat. No. 5,642,108 to Gophert et al. and U.S. Pat. No.5,689,253 to Hargreaves et al.) or to palm presses, as in U.S. Pat. No.5,017,030 to Crews. However, for a multi-touch surface, reaching thethumb for modifier keys poses the same drift exacerbation problems asreaching by the pinky, and palm touches should be ignored to encouragehand resting. Thus there exists a need in the multi-touch and chordkeying art for alternative methods to activate modifier keys withoutdrawing any fingers away from the row.

BRIEF SUMMARY OF THE INVENTION

In its primary aspect, this invention introduces four-fingertip modifierchords to eliminate the hand twist and reach traditionally required toactivate modifier keys. Simultaneously dropping the four long fingertipsof a hand into a modifier zone on or near the home row keys applies theShift modifier to subsequent typing or pointing input so long as anyfinger from the modifier chord remains touching the surface. Typically,then, the modifier will apply to activity by the opposite hand, but thepresent invention also lets a hand modify its own typing, therebyallowing capitalization of whole words, if at least one of its modifierchord fingertips remains touching as others lift to strike nearby keys.The four-fingertips (excluding the thumb) chord is preferred for thisrole because it is the easiest to drop and hold on the surface besidesthe five-finger chord, which must be reserved for hand resting.

Since the four-fingertip chord is also preferred for window scrolling,and since it is often a prelude to dropping the thumb into the fullfive-finger hand resting chord, the present invention takes specialprecautions to prevent accidental modifier activation. The modifierpress signal is not sent to the host computer immediately upon detectionof the modifier chord touchdown. The modifier press will only be sent,commiting the modifier, upon detection of modifiable input activity byother than the thumb of the modifying hand. Modifiable input activitycan include any user action that produces a keypress, pointing,dragging, clicking or other command for the host computer, but does nottypically include resting touches that cause no signals to be output.Any touch by the modifying hand's thumb detected before commit willimmediately cancel the modifier chord, effectively turning it into ahand resting chord. Such thumb touches after the modifier press orcommit need not permanently cancel the modifier. However, if these thumbtouches represent editing keys such as Space or BackSpace keypresses,the Shift modifier signal may release temporarily while the thumb key istransmitted since the typist is most likely just erasing or putting anormal space between two capitalized words.

Restricting the Shift modifier chord to a zone along home row encouragestypists to return their hands to the home typing position. Furthermore,this allows a Ctrl modifier zone to be established along the row of keysabove home row, an easy stretch from home row. A third modifier zone canbe established along the row of keys below home row for rarer modifierssuch as Alt, Windows, Open Apple, or Meta. Even a fourth modifier zoneis possible approximately two key rows below home row. Note that all ofthese modifier zones can be reached through straight flexion orextension of the fingers from their home row position-absolutely notwisting or rotation of the wrist or fingers is necessary.

According to the present invention, multiple modifiers are activated bythe same hand simultaneously when the fingertips of the modifier chordare clearly distributed into different modifier zones. To compensate forthe natural arch in a row of fingertips, the vertical offset of eachfinger is measured relative to the home row key the finger normallyrests upon. Accidental activation of a multiple zones is prevented bychecking for a minimum interval between the vertical offsets offingertips in different zones. If this condition is not met, the averageof the vertical fingertip offsets is used to choose a single modifierzone. Distributing the fingertips into different zones does imply somefinger twisting, but does not cause as much hand drift as reaching formultiple modifier keys on the periphery of the key layout.

In an alternative embodiment of this invention, different modifiers canbe activated in a manner independent of any zones or the overall handposition on the surface. Rather, they are distinguished by differenthorizontal and vertical separations between the four fingertipsperforming the chord. Shift, for example, might be activated by thenormal relaxed placement of four fingertips in a row with about 2 cm(¾″) separating the fingertip centers. Ctrl would then be activated byplacing the fingertips stretched along the row with an average 3 cm(1⅛″) separation between them. A third modifier could be activated bysplitting the fingertips vertically into two rows a couple cm apart.This aspect of the invention is most useful for non-typing situationswhere hand motions are not focused around a default position along homerow. With this aspect, a hand can, for instance, apply differentmodifiers to mouse clicking activity on the opposite hand without havingto reposition itself within certain modifier zones.

A primary objective of the present invention is to provide an apparatuscapable of detecting four-fingertip modifier chords that obviate theawkward pinky or thumb reaches previously needed to strike and holdmodifier keys.

Another objective of this invention is to prevent spurious modifierchord activation when the user is slowly relaxing into a hand restingchord, but does not initially have the thumb on the surface.

A further objective of this invention is to allow use of the Shiftmodifier chord for capitalization across words without applying theShift modifier to intervening Space or BackSpace key activations bythumbs.

Yet another objective of this invention is to establish differentmodifier zones across, above, and below the home row of keys that can beutilized to apply different modifiers.

A further objective of this invention is to support simultaneousactivation of multiple modifiers with the same hand when the fingertipsof the modifier chord are clearly distributed among different modifierzones.

Another objective of this invention is to support selection of differentmodifiers from the relative arrangement of fingertips within a modifierchord rather than their placement within any particular zone on thesurface.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block level diagram of a multi-touch system with typing,chord motion, and modifier chord recognition according to a preferredembodiment of the present invention.

FIG. 2 is a diagram of a split QWERTY key layout with prior art modifierkeys that are struck by the pinky or thumb.

FIG. 3 is a diagram of a split QWERTY key layout with a different chordmodifier zone overlapping each row of keys according to a preferredembodiment of the present invention.

FIG. 4 is a data flow data diagram showing the relation to the processesand data that the present invention uses to detect and apply modifierchords in its preferred embodiment.

FIG. 5 illustrates the contents of the touch data structure used tostore touch location, identity and timing and, when formed into a chain,represent a sequence of touches.

FIG. 6 illustrates the contents of the data structure used to keep trackof a modifier chord's state through its life cycle of being canceled orapplicable and eventually lifted.

FIG. 7 illustrates the configuration parameters used in the preferredembodiment to represent a single modifier zone.

FIG. 8 is a flowchart of the process that detects new modifier chords.

FIG. 9 is a flowchart detailing computation of fingertip verticaloffsets from the default fingertip locations.

FIG. 10 is a flowchart of the preferred embodiment of the process thatdetects which modifier zone(s) a chord is selecting.

FIG. 11 contains touching timing diagrams showing preferred systemresponses to thumb touches during modifier chord performance.

FIG. 12 contains touch timing diagrams that demonstrate sustain of amodifier chord throughout typing touches by the modifier chordfingertips.

FIG. 13 contains touch timing diagrams for cases in which a modifierchord should be allowed to apply to touch activity that actually occursslightly before its touchdown or after its liftoff.

FIG. 14 is a flowchart of the process that detects modifier chordcancellation and/or liftoff according to the present invention.

FIG. 15 is a flowchart of the process that searches for modifier chordsor keys applicable to modifiable input activity at a given time.

FIG. 16 shows several distinct fingertip arrangements that can beconfigured to select and apply different modifiers in an alternative tomodifier zones.

FIG. 17 illustrates alternative configuration parameters that can encodea template for a recognizable arrangement of fingers within a modifierchord.

FIG. 18 is a flowchart of the alternative modifier selection processthat finds the fingertip arrangement template that most closely matchesthe fingertip arrangement within a performed chord irrespective ofabsolute hand position on the surface.

DETAILED DESCRIPTION OF THE INVENTION

In the preferred embodiment, the typing recognition methods of thisinvention are utilized within a multi-touch system like that shown inFIG. 1. The sensor scanning hardware 6 detects touches by fingers 2 onthe touch surface 4. The proximity image formation 8 and contacttracking and identification 10 modules determine the touch timing andsurface coordinates and report these to the typing recognizer 12. Thetyping recognizer decides which keys the user intended to press andtells the host communications interface 16 to send those keys to thehost computer 18. The chord motion recognizer module 14 that interpretslateral sliding of multiple fingers as pointing or gesture input andeffectively disables the typing recognizer for such touches. Thesynchronization detector 13 searches for simultaneous presses orreleases of multiple fingers, thereby aiding in detection of chordslides, chord taps, resting hands, and, for the purposes of thisinvention, modifier chords. Prior art embodiments of all modules in FIG.1 except the MODCHORD subdivisions 15 are described in related U.S. pat.app. Ser. No. 09/236,513 by Westerman and Elias. That application isincorporated herein by reference in its entirety. It disclosestechniques for detection of chord taps and generation of single commandsor button clicks therefrom. It also discloses methods to detect andapply conventional modifier key touches. However, unlike the presentinvention, that application does not teach recognition of modifierchords nor associated techniques for selecting modifier types accordingto the zone a chord falls within or the template arrangement bestmatching its finger arrangement, canceling the chord in response tothumb resting, committing a modifier chord upon reception of subsequentmodifiable input activity, nor sustain of modifier signals throughtyping by both hands until no fingers from the modifier chord remaintouching. The detection 56, cancellation 60, and application 62processes embodying the improvements of the present invention are widelydistributed across the typing recognizer 12, synchronization detector13, and chord motion recognizer 14 modules so as to efficiently applymodifier chords to the other input activities that these modulesrecognize. Thus these modifier recognition processes are collectivelyrepresented in FIG. 1 by the MODCHORD subdivision 15 of each module. Theexact relation of the modifier chord recognition processes to eachlarger recognition module will be made apparent from the followingdetailed description. U.S. pat. app. Ser. No. 09/681,146 by Westermanfurther describes an improved typing recognizer 12 that compares touchgeometry to key sequence candidate geometry, but its improvements do notbear directly on modifier chord recognition.

Those skilled in the art will recognize that the modifier chordrecognition method disclosed herein could be utilized with any sensingdevice that accurately reports the lateral position of multiplefingertips on a surface. Likewise, the modifier chord recognitionsoftware need not reside within the sensing device. It could just aseasily execute within the host computer system, or the host computersystem and sensing device might be combined such that the samemicroprocessor executes finger tracking, modifier chord recognition, anduser application software. Those with ordinary skill in the art willalso be aware that some keyboard interfacing protocols useedge-signaling of key activation state while others use level-signaling.For instance, keyboards with the legacy PS/2 interface forIBM-compatible PCs will transmit a press keycode only upon initialactivation of a modifier keyswitch and will send a corresponding releasekeycode immediately after the finger lifts off the switch. Thus thepress/release keycodes are only transmitted at edges or transitions inthe state of the keyswitch. Keyboards communicating via the more recentUSB (Universal Serial Bus) protocol use level-signaling: as long as akeyswitch is depressed, the keyboard regularly and repeatedly sends thecorresponding keycode to the host computer. There are no distinct pressand release keycodes. This disclosure will use the edge-signaling,press/release terminology throughout to describe transmission ofmodifiers to the host computer 18, but it will be apparent to those ofordinary skill in the art how any edge-signaling implementation of thehost communications interface 16 can be converted to level-signaling andremain well within the scope of this invention.

The key layout illustrated in FIG. 2 exemplifies prior art placement ofmodifier keys 20, 21, and 22 such that they can be reached by the thumbor pinky fingers. This diagram shows a QWERTY layout split into left 23and right 24 halves such that Ctrl and Alt modifier keys 22 can beplaced between the split halves within easy reach of either thumb.Non-split layouts are more likely to place Ctrl and Alt modifier keys 21on the bottom row of the layout where they are most likely to beoperated by the pinkies. Whether split or not, most layouts keep theShift modifier keys 20 diagonally below the home row key (‘A’ or ‘;’)that the pinky normally rests upon. Though these Shift keys 20 are notthat far from home row, the pinky motion needed to reach and hold themdown is particularly awkward.

In contrast to prior art FIG. 2, FIG. 3 shows the chord modifier zones30-34 and 35-37 of the present invention as different hatches rangingacross each row of the key layout. In an actual product, the differentmodifier zones would be indicated on the touch surface by differentbackground colors rather than different hatches. According to thepreferred embodiment of the present invention, simultaneous touchdown offour fingertips within one of these zones will cause subsequent touchactivity to be modified just as if a modifier key was being held as longas the modifier chord is not fully lifted or canceled by an improperthumb touch. Notice that a hand resting on the home row ‘ASDF’ or ‘JKL;’keys can activate any of its modifier zones by pure finger flexion orextension, eliminating the awkward twists needed to reach conventionalmodifier keys. Notice also that the modifier zones are not horizontallyrestricted to the ‘ASDF’ or ‘JKL;’ key columns, so the typist need notworry about horizontal hand alignment when performing a modifier chord.The key layout improved with modifier zones may retain the conventionalShift modifier keys 20 near the pinky for the convenience of novices whohave not yet learned to perform modifier chords, but experience with theinvention has shown that typists rarely use these conventional modifierkeys once they have learned the more convenient modifier chords. Sincecapitalization is needed so frequently, the modifier zones 31 and 36along the left and right home row of keys are preferably assigned theLeft Shift and Right Shift modifiers. The modifier zones 30 and 35 abovehome row are preferably assigned the Left Ctrl and Right Ctrl modifiers,and the zones 32 and 37 directly below home row are preferably assignedthe Left Alt and Right Alt modifiers. Modifier zones 33 and 38 can beassigned any remaining modifiers such as Meta, Diamond, Windows, or Openapple supported by the host computer 18's particular operating system.Those skilled in the alternative key layout arts will realize thatmodifier zones can be established just as easily over Dvorak, non-split,or non-English key layouts. Also, those skilled in the foreign keyboardarts will realize that the modifier zones could alternatively beassigned to Asian language modifiers like Kana or Kanji and remain wellwithin the scope of this invention.

In the preferred embodiment of the present invention, modifier chordrecognition is split into three processes as shown in FIG. 4. Process56, detailed in FIG. 8, detects new modifier chords by monitoring theincoming touch sequence 50 for touchdowns by the four fingertips of onehand that are simultaneous and also lie within one of the zones orarrangements specified by the configuration data 54. This process onlyneeds to execute each time a new finger touch is detected and appendedto the touch sequence, not for image frames from the sensor scanner 6that contain no new surface contacts. The incoming touch sequence 50consists of a chain of touch data structures 80 (detailed in FIG. 5)ordered by touchdown times 85. The configuration data structures 100 and520 that store the modifier zones or arrangements 54 are detailed inFIG. 7 and FIG. 17. If it detects a new modifier chord, process 56 willallocate a corresponding modifier state data structure 90 (detailed inFIG. 6 ) and appended it to the modifier chord state chain 58. Theliftoff/cancellation detection process 60 (detailed in FIG. 14 ) checkseach new image frame or sensor array scan for absence of all fourfingertips or presence of the thumb from any hand with a modifier chordpending in the state chain 58. Upon detection of one of theseconditions, the liftoff/cancellation process 60 uses feedback path 68into the modifier state chain 58 to record either the chord liftoff time94 or chord cancellation 96. It may also send corresponding modifierrelease signals to the host computer 18 through the host communicationinterface 16. The third process 62 searches the modifier state chain 58for uncancelled modifier chords roughly coincident with modifiable inputactivity. If this process finds any applicable modifier chords whosepress signals have not already been sent, it will cause the hostcomputer interface 16 to send their modifier press signals to the hostcomputer 18. This search process 62 is typically called by the typingrecognizer 12, synchronization detector 13, or chord motion recognizer14 with the timestamp 64 of the modifiable input activity right beforesuch activity itself is transmitted through the host communicationinterface 16.

FIG. 5 lists basic parameters needed for each touch data structure 80 tosupport detection of new modifier chords 56. A ring or chain of suchdata structures ordered by touchdown time 85 represents a touch sequenceby one or more fingers. Since palm touches are to be ignored by allrecognition processes, they can be left out of the touch sequence 50.Each touch data structure 80 must contain the touch's x and y surfacecoordinates 82 as reported by the touch sensors 6. These should estimatethe center of the touch, which for proximity or pressure sensors istypically computed as the centroid of fingertip flesh contacting thesurface. The y offset from default finger location 84 will be computedin FIG. 9 to improve the accuracy of modifier zone selections. To helpdetermine whether a touch sequence represents the four fingertips of amodifier chord, each touch data structure should have a copy of the handand finger identity 84 estimated for the touch by the contact trackingand identification module 10. To help detect synchronization of multipletouches, the touch data should also include the finger touchdown time85, also reported by the contact tracking and identification module 10.While the touch liftoff time 86 is a useful indicator of individualmodifier key release, the modifier chord liftoff time 94 is not derivedfrom the individual liftoff times 86 of the touches originally formingthe chord. This is because in its preferred embodiment the presentinvention beneficially allows individual fingertips of a modifier chordto lift temporarily and touch again to type while other modifier chordfingertips remain touching, as is shown in FIG. 12. These temporaryfinger lifts will establish the touch liftoff times 86 for the originaltouches forming the chord but may not be indicative of final chordliftoff since the temporarily lifted fingers may touch again to sustainthe chord while other modifier fingertips lift to type, and so on.

FIG. 6 lists the parameters that keep track of a modifier chord's state90 once it is initially detected by process 56. The modifier type bits91 encodes which modifiers, e.g. Shift, Ctrl, or Alt, the chord'sfingertip placement has selected. The contents of these bits will bedetermined by either the modifier zone selection process of FIG. 10 orthe fingertip arrangement selection process of FIG. 18. The hand field92 indicates which hand performed the modifier chord. The chordtouchdown time 93 is set by the new chord detection process 56 as theminimum of the touchdown times 85 of the original four fingertip touchesfirming the chord. As discussed previously, the chord liftoff time 94should not be computed as the maximum of the original touch liftofftimes 86. Instead, the liftoff and cancellation check process 60, FIG.14, will set the chord liftoff time 94 as the first time after chorddetection that none of the four fingertips from the chord are found tobe touching the surface. The canceled flag 96 is set by the liftoff andcancellation check process 60 if the chord fingertips slidesubstantially or a thumb from the chord's hand touches before the chordas committed. When set, this flag causes the applicable modifier searchprocess 62 to ignore the modifier chord. The committed flag 97 is setonce a modifier chord is actually applied by the applicable modifiersearch process 62 to outgoing typing or clicking activity. Once set,this flag prevents the chord from being canceled except when fullylifted.

FIG. 7 shows the data structure 100 and parameters used to configureeach modifier zone. Here, the modifier type bitmask 102 is a set offlags indicating which of the Shift, Ctrl, Alt or other modifiers areassigned to the zone. Note that this bitmask implementation allowsmultiple modifiers, such as Shift Ctrl, to be assigned to the same zoneif the user so desires. The hand field 104 can restrict the zone to theleft 23 or right 24 half of the key layout, but no other horizontalalignment restriction is necessary. The min 106 and the max 108 Y offsetfrom home row determine the vertical range of the modifier zone. Notethat no extra configuration parameters are necessary to arch the zonesalong each row, should the key rows be so arched, because the zoneselection process of FIG. 10 will compare these Y offsets with eachfingertip's vertical offset relative to the position of the home row keythe finger normally rests upon, also know as the fingertip's defaultposition. Thus the modifier zone will automatically arch to match anyarch in the home row key locations.

The new modifier chord detection process is shown in FIG. 8. Thisprocess begins whenever a new finger touch is detected by the contacttracking and identification module 10. Step 150 increments the latesttouch index n and stores the touch's parameters 80 at the n th locationof the touch sequence array T[ ]. Steps 152, 154, and 156 scan backwardm touches in the touch sequence trying to find the largest synchronizedsubsequence that includes the new touch T[n]. Decision diamond 154judges synchronization by testing whether the m th previous touch T[n−m]contacted the surface within a synchronization interval of about 60milliseconds of the new touch T[n]. Note that the typing recognizer 12should not generate signals to the host corresponding to an individualtouch over a key until sufficient time has passed without subsequenttouches on the same hand that this synchronization detection loop can becertain that the touch is not synchronized with later touches. Thus thetyping recognizer 12 must delay key output about 60 ms from fingertouchdown or be prepared to erase or undo keys from touches later foundto be part of a chord. Once decision diamond 154 finds a previous touchtoo old to be synchronized with T[n], it passes on the largestsynchronized subsequence as T[n−m+1] . . . T[n].

Decision diamond 158 then examines the finger and hand identity 84 ofeach synchronized touch looking for a combination of identifies from onehand that matches any combination allowed for modifier chords. In thepreferred embodiment, only the 4 fingertip combination, index, middle,ring and pinky, excluding the thumb, is used for modifier chords. Toprevent duplicate detection of the same modifier chord, decision diamond158 must require the newest touch T[n] to be one of the modifier chordfingertips. Otherwise, any synchronized touches intervening from thehand opposite a modifier chord combination do not affect the modifierchord, but the modifier chord may eventually apply its modifiers to themas in touch 370 of FIG. 13. If the synchronized subsequence does notcontain a modifier chord combination from either hand, the processreturns 160 until the next new touch warrants a renewed detectionattempt.

Assuming a modifier chord combination from one hand is found within thesynchronized subsequence, block 162 forms a touch array MT[ ] indexed byfinger identity containing only the modifier chord touches. As furtherdescribed in FIG. 9, block 162 also computes each touch's verticaloffset from its corresponding home row key or default location. Block164, further described in FIG. 10, checks whether the chord has beenperformed within any of the established modifier zones. In analternative embodiment, block 164 may check the arrangement offingertips within the chord as further described in FIG. 18. Decisiondiamond 170 ends new modifier detection through step 168 if the chordmatches none of the established zones or arrangements. Otherwise, step172 allocates a new modifier state 90, setting the modifier type 91according to that of the selected zones or arrangement, setting themodifier hand 92 to be detected chord's hand identity, and setting thechord touchdown time 93 as the minimum of the touchdown times 85 of thesynchronized touches forming the chord, and setting the chord liftofftime 94 to 0 pending full liftoff detection. Step 172 also appends thisnew modifier state to the state chain 58 that may already contain statefrom other chord or key modifiers being held by the opposite hand. Newmodifier detection returns at step 176, and responsibility for cancelingthe modifier chord or applying it to modifiable input activity passes toprocesses 60 and 62.

FIG. 9 shows the details of vertical offset computation block 162. Step200 starts the sync index s at the oldest synchronized touch and loops210 through the synchronized subsequence. Decision diamond 204 separatesout the modifier chord touches from any opposite hand touches. For theconvenience of the modifier zone or arrangement matching process 164,step 206 stores each modifier chord touch into the touch array MT[ ] inorder of their finger identity 84. Step 208 computes each modifiertouch's vertical offset from the default or resting location of itscorresponding finger. For key layouts such as 23 and 24 with an archacross the home row keys, these default locations should be thelocations of the ‘ASDF’ and ‘JKL;’ keys. These vertical offsets help themodifier zone matching process 164 efficiently compensate for thenatural arch across the fingertips. Decision diamond 202 breaks the loopand returns through 212 once the whole synchronized subsequence has beenprocessed.

FIG. 10 discloses the preferred embodiment of block 164, a search formodifier zones 100 that the modifier chord fingertips lie within. Thearray Z[ ] of configured zones is assumed to contain only zones for thegiven modifier chord's hand, and these zones are assumed to be orderedwithin the array from farthest below home row to farthest above. In thisembodiment, the loop of steps 250-276 will attempt to accumulatemodifier types from each zone that any of the modifier chord fingertipslies within. If, however, decision diamond 262 finds that the fingertipsare not clearly spaced across zones, i.e. if two or more fingertips arevertically bunched together yet straddling the border between two zones,it will direct the loop of steps 280-288 to find the single zone thatthe average of the fingertip offsets falls within. Step 250 clearsselected_mods, the variable whose bits will accumulate the selectedmodifier types. Step 252 clears the modifier zone index i. Step 254initializes the max vertical offset for fingertips found in the i thzone, Z[i].found_ymax, to the zone's minimum vertical boundaryZ[i].range_ymin. Step 256 initialized the modifier fingertip index f to2, representing the index finger. Decision diamond 260 checks whethertouch MT[f]'s vertical offset from default is within zone Z[i]'sboundaries. If not, decision diamond 270 checks whether all fingertiptouches including the pinky have been examined, and if not step 272advances the fingertip index. If touch MT[f] is within Z[i]'s verticalrange, decision diamond 262 checks whether it is vertically separated byat least zone_safety_sep from any touches already found in the zonebelow. zone_safety_sep should be set to about 1 cm or ⅜″. If the touchis clearly separated form any in the zone below, step 264 will bump upZ[i].found_ymax as necessary. Step 266 accumulates the modifier types 91assigned to Z[i] into the selected_mods bits with a bitwise ORoperation. Once all fingertips have been checked against the i th zone,decision diamond 274 will check whether all zones have been tested, andif not advance to the next zone through step 276. If the loop 254-276gets through all zones without decision diamond 262 finding a fingertipseparation violation, step 278 will return the accumulated modifiertypes selected_mods to decision diamond 170 of FIG. 8.

If decision diamond 262 detects an interzone fingertip separationviolation, the zone index i and selected_mods are reset at steps 280 and282. Step 284 computes from MT[ ]the average avg_yoffset of all fourfingertips' vertical offsets. Decision diamond 286 checks whether thisavg_yoffset is within the range of zone Z[i]. If not, step 290 advancesthe zone index to the next zone until either decision diamond 292 findsall zones have been exhausted or decision diamond 286 finds a matchingzone. Assuming avg_yoffset falls within the vertical range of one of thezones, step 288 assigns selected_mods the modifier type(s) of that zone,and step 278 returns these. Note that step 278 will return zero if thefingertips are not within range of any zone.

Before describing in detail the liftoff/cancelation detection process 60and the search for applicable modifiers 70, it will be helpful to definetheir preferred behavior with the diagramed typing examples of FIGS.11-13. These timing diagrams display the touchdown (falling edge) andliftoff (rising edge) timings for particular fingers 84 from both hands,where LF5 denotes the left pinky, LF1 the left thumb, RF2 the rightindex finger, and so on. Fingers not shown in an example can be assumedto be lifted throughout. All synchronized chord touches 300 are assumedto fall within a Shift modifier zone. The vertical dotted lines 93 and94 identify the chord touchdown time and liftoff time, respectively.Keys transmitted to the host by the typing recognizer 12 in response toasynchronous touches are denoted by the key's symbol encircled. The keysymbol is only capitalized if the Shift modifier chord applies to it.The slight space drawn between each touchdown and the corresponding keycharacter circle simply indicates that the typing recognizer must waitabout 60 ms after each touchdown before sending a keypress to ensurethat the touch will not be part of a chord. The left and right ends ofSHIFT ellipses 318, 319, 338, 339, 348, 378, and 388 in the MODIFIER 66row demarcate the modifier press and release signals that should be sentto the host computer. Thus the Shift modifier is being applied throughthe duration of each ellipse. The bottom two rows of each timing diagramindicate setting of the modifier state canceled flag 96 or committedflag 97 in response to certain touches.

FIG 11A shows typing before, during, and after performance of a lefthand Shift chord. The initial ‘a’ key touch 310 clearly precedesmodifier chord touchdown 93 and so should not be capitalized. Noticethat the Shift modifier is not sent to the host, committing the chord469, until the ‘J’ touch 311 needs to be sent to the host computer, wellafter modifier chord touchdown. Committing of modifier chords should bethus delayed until transmission of subsequent modifiable activity (the‘J’ key touch in this case) to allow for the possibility that the 4fingertips will begin sliding, suggesting that the typist is actuallytrying to scroll, or be supplemented with a thumb touch, suggesting thetypist is just resting the hand sloppily. Thissubsequent-touch-activity-dependence of modifier chord committing isunique and novel, as the chord and key taps of the related art commiteither on liftoff or sufficient touching time of the tapping fingersthemselves, while chord slides for pointing and command gestures commitupon significant lateral motion of the involved fingers. The ‘I’ touch312 is also capitalized, but the Shift modifier is temporarily releasedwhile the BackSpace key touch 302 is sent to the host, and then Shift ispressed again at 319 in time for the ‘U’ key touch 313. Decision channel463 of FIG. 15 will handle such temporarily release of thumb editingkeys such as BackSpace and Space under the assumption that typistsusually intend to make quick, unshifted edits but are too lazy to liftand retouch the Shift chord before and after the thumb key activations.The ‘n’ key touch 314 clearly follows modifier chord liftoff, andtherefore should not be capitalized.

FIG. 11B demonstrates the desired system behavior when the modifyinghand's thumb touches down before typing or clicking activity amongst theother fingers has a chance to commit the modifier chord. As before, thetouches 320 and 323 before and after modifier chord performance do notget capitalized. However, the ‘i’ and ‘n’ key touches 321 and 322concurrent with the modifier chord do not get capitalized either becausethe thumb touch 302 preceding them causes the modifier chord to becanceled 404. Whether the thumb touch lands properly on and activatesthe BackSpace key or whether it lifts back off before the ‘i’ and ‘n’touches is not important. The thumb touch 302 is quite likely the resultof sloppy hand resting, and should cancel the modifier chord lestsubsequent typing or clicking be shifted unintentionally and have to beundone. Decision diamond 402 of FIG. 14 will implement cancellation dueto resting thumb touches.

FIGS. 12A and 12B demonstrate how a modifier chord can be sustainedthrough typing by its fingers on its half of the key layout so thatwhole words can be capitalized from a single modifier chord touchdown.In FIG. 12A, the ‘K’key touch 330 on the opposite hand quickly commits469 the modifier chord and the ‘I’ touch 331 follows. To obtain acapitalized ‘D’ and ‘S’, the typist need only lift the middle and ringfingertips of the modifier chord and set them down 332 and 333 on thosekeys one at a time. The typing recognizer must be configured to generatekeys from asynchronous touches without waiting for touch liftoff forthis to work well, as at least one of the resumed touches 332 and 333will need to sustain one of the resumed touches 332 and 333 will need tosustain the modifier chord while the pinky and index of the modifierchord temporarily lift to touch the ‘A’ 335 and ‘T’ 336 keys. Chordliftoff 94 does not register and cause release of the modifier 349 untilthe first moment when none of the modifier fingertips are touching. Theright thumb Space key touch 334 invokes temporary modifier release as inFIG. 11A. FIG. 12B is similar to FIG. 12A except that the modifier chordis performed by the right hand and committed 469 by a temporary liftoffand ‘P’ touchback 340 of one of its own pinky fingertip. Touches 341-343walk through the other modifier chord fingertips ending with a ‘T’ touch344 by the opposite hand to spell ‘POINT’. This demonstrates that amodifier chord need not always be committed by the opposite hand, whichcan be useful in situations where only one hand is available forinteraction with the touch surface.

Conventional mechanical keyboards never apply modifier keypresses tokeys barely preceding the modifier keypresses or just following themodifier key release. The comparative lack of keyswitch action or strokefor touch surfaces lessens the typist's control over the timing betweenmodifier touch and the touches to be modified. FIGS. 13A and 13Bdemonstrate cases where it is helpful for the system to tolerate someinaccuracies in the timing between modifier chord touches and modifiablekey touches. The same techniques also apply to individual modifier keytouches on a surface. In FIG. 13A, the touch 370 actually precedes themodifier chord touchdowns 300 but is roughly synchronous with them.Assuming the modifier chord is detected before the key from touch 370gets sent to the host, the modifier chord should apply to and capitalizethe ‘T’. Such handling of key touches that very slightly precedemodifier chord touches is important for typists who get in the habit ofsimultaneously touching both the modifier chord and the opposite hand'skey to be capitalized. The system will implement it through thesync_slack term of decision diamond 456 in FIG. 15. The ‘T’ iscapitalized again for touch 371 but not for ‘t’ touch 372, which followschord liftoff. FIG. 13B shows a brief tap of a modifier chord withoutany coincident key touches. However, ‘T’ touch 382 does follow shortlyafter chord liftoff. This suggests the typist was intending tocapitalize a single key but, in performing the modifier chord soquickly, synchronized it poorly with the key touch 382 on the oppositehand. It is desirable for the system to compensate by applying themodifier chord in this case even though this means committing 469 andsending the modifier press signal 388 after the chord has actuallylifted 94. This is implemented with a non-zero lift_slack in step 464and decision diamond 466 of FIG. 15.

FIG. 14 details process 60, thumb cancellation and chord liftoffdetection. Step 400 denotes that whenever any finger touchdown orliftoff is detected on the surface, this process must be repeated foreach modifier chord state 90 in the state chain 58. The combination ofdecision diamonds 401 and 402 causes the chord to be canceled in step404 if a thumb touch by the modifier chord's hand is detected before thechord's committed flag gets set. Decision diamond 402 may also decide tocancel a chord if excessive sliding is detected amongst the chord'sfingertips. If no such canceling thumb touch is detected, decisiondiamond 406 checks whether chord liftoff has previously been detected.If not, decision diamond 408 checks the latest sensor image frame 52 forany touches by the modifier chord fingertips. If none are found, step410 sets the chord liftoff time 94 to the timestamp of the latest sensorimage. If liftoff was just detected and decision diamond 412 finds thatthe modifier types 91 for this modifier state MS are currently pressedfor the host, step 414 instructs the host communications interface 16 tosend the corresponding modifier release signals. Steps 404 and 410, ifexecuted, thus constitute the feedback path 68 from this process 60 tothe modifier state chain 58.

FIG. 15 details the search for applicable modifiers 62 that shouldalways execute just before the typing recognizer 12, synchronizationdetector 13, or chord motion recognizer 14 transmit any typing,pointing, or clicking signals to the host computer 18. These moduleswill pass to the search process the modifiable tstamp parameter, set tothe current system time for pointing activity but otherwise to theoriginal touchdown time of the key or click chord touches. Step 450clears the applicable_mods variable, which will accumulate the bitmasksof the types of modifiers found applicable. Step 452 initializes themodifier state reference MS to the most recent modifier state 90 in thechain 58. Decision diamond 456 causes canceled 404 chords to be excludedfrom the applicable modifier search. At this point the modifier statefor a canceled chord can, if memory resources are scarce, be entirelyremoved from the state chain 58.

Assuming the chord has not been canceled, decision diamond 456 ensuresthat modifiable_tstamp does not precede the chord touchdown time 93 bymore than a sync_slack of about 60 ms. If the chord more or lessprecedes the modifiable activity, step 460 establishes a 0 ms defaultlift_slack. If the modifier chord has only been touching briefly and isnot yet committed, decision diamond 462 will cause step 464 to establisha lift_slack of about 100-150 ms, allowing for late modifiable touchesas in FIG. 13B. If the chord has been touching a few seconds or hasalready been committed by other key touches, decision diamond 463 checkswhether the modifiable activity is a thumb keypress, and if so, shouldprevent the modifier chord from applying at least if the chord is aShift modifier. This will also cause temporary Shift modifier releaseduring thumb key transmission as in FIG. 11A and FIG. 12A. Note thatthis-check is not done for briefly touching chords that have notcommitted yet so as to allow the typist to perform, for example, asingle Shift BackSpace macro command using a modifier chord from thehand opposite the thumb key.

The remaining steps ensure that the modifier chord did not liftsubstantially before modifiable_tstamp. This is certainly true if themodifier chord is still touching and has not yet been assigned anon-zero liftoff time, as detected by decision diamond 465. If the chordhas already lifted, decision diamond 466 will need to compare the chordliftoff time 94 with the modifiable_tstamp less the lift_slack todetermine whether the modifiable activity occurred before or withinlift_slack milliseconds after chord liftoff. Assuming the modifiableactivity is found to be sufficiently coincident with the modifier chord,step 469 will set the modifier state's committed flag 97, and step 468will accumulate the modifier types selected by the applicable modifierstate. Thus step 469 constitutes the main feedback path 70 from theapplicable modifier search process 62 to the modifier state chain 58.Decision diamond 471 checks whether any older modifier states that mightbe applicable are left in the state chain 58, and if so step 470continues the search on the previous modifier state. Once the entirestate chain has been examined, the process returns at step 472 themodifier types from the modifier chords found applicable. The hostcomputer interface 16 will compare these newly applicable modifier typesto a bitmask of the modifier types already pressed and send additionalmodifier press or release signals to the host computer as necessary.

Those skilled in the chord keying art will realize that this chordmodifier recognition method could easily be adapted to keyboards withconventional mechanical keyswitches, assuming the keyswitch matrix waswired such that the keyswitch scanner circuitry could reliablydistinguish coincident presses of four home row keys forming the chordas well as one other key to be modified. However, such an adaptationwould not be terribly advantageous because the effort needed to pressfour mechanical homerow keyswitches simultaneously is just about asstraining as the awkward pinky reach for a conventional Shift key.However, when modifier chords are utilized with a proximity sensingmulti-touch surface as in the preferred embodiment, holding the chordrequires no effort giving modifier chords significant advantage over thepinky reach for a conventional Shift key.

While modifier zones beneficially allow simultaneous selection of anycombination of up to four different modifier types by a single hand, thetypist must be careful of hand alignment with respect to the zones tooperate them accurately. On surfaces without a home row of keys toencourage hand alignment, or when the user is primarily pointing ratherthan typing, an alternative, hand-position-independent method to selectdifferent types of modifiers is desirable. FIG. 16 shows six sets ofchord touches that are distinguishable not by their finger combinationor alignment with respect to the key layout, but by the relativearrangement of the four fingertips index 502 through pinky 505 withinthe chord. The arrangements shown would most likely arise from righthand chord performances; left hand chord performances would produce themirror image of these arrangements. Each arrangement can be assigned adifferent modifier type to allow a wide range of modifier selectionsanywhere on the touch surface. The arrangement of FIG. 16A is mostrelaxed and is thus preferred for the Shift modifier. The arrangement ofFIG. 16B is preferably mapped to Ctrl, FIG. 16E to Alt, FIG. 16C to CtrlShift, and the remaining arrangements to any other modifiers useful on aparticular host computer. The crosshairs 500 indicate the centroid ofeach fingertip arrangement and will be used as the coordinate originwhen storing the arrangements as templates.

FIG. 17 shows a configuration data structure 520 suitable for storingfingertip arrangement templates and associating them with particularmodifier types. Thus configuration data structure 520 is an alternativeto the modifier zone embodiment 100 of configuration data 54. It has amodifier type bitmask 102 and modifier hand source 104 equivalent to themodifier zone data structure 100. Each data structure 520 encodes one ofthe fingertip arrangements from FIG. 16 or the like as a template offour x offsets 526 and four y offsets 528 from the chord center 500.Encoding them as offsets from the centroid 500 of the 4 fingertiplocations will help template matching in FIG. 18 be neutral to absolutehand position on the surface.

The pattern matching process of FIG. 18 selects the template arrangementthat best matches the measured arrangement of fingertips performing amodifier chord. This arrangement matching process is thus an alternativeto FIG. 10's modifier zone selection embodiment of block 164. An arrayAR[ ] of the template data structure 520 constitutes the data 54. Step550 clears selected mods, the variable whose bits will hold the selectedmodifier types. Step 552 initializes min_sumdist2, the variable thatwill hold the lowest match error of evaluated templates, to a very largenumber. Step 555 compares the average or centroid, avg_fpos, of the fourfingertip locations from the MT[ ] modifier touch array. Step 554 clearsthe arrangement template index i. Step 556 clears the squared errorsum_dist2 for the current template AR[i]. Step 558 initializes thefinger identity index to 2 indicating the index finger. Steps 560-564then loop through to the pinky accumulating the squared error betweeneach fingertip's offset from chord center and the corresponding templatepoint's offset from chord center. Decision diamond 566 checks whetherthe squared error for template AR[i] make it the best match so far, andif so step 568 updates min_sumdist2 and step 570 stores template AR[i]'smodifier type 522 as the modifier selected so far. Decision diamond 572determines whether all template arrangements have been tested, and ifnot step 574 advances the index i to the next template. Once allarrangements have been tested, step 576 will return the modifier type522 of the template 520 which most closely matches the measuredarrangement of modifier touches MT[ ].

Those with ordinary skill in the art will be able to adapt the presentinvention to use different finger combinations such as three fingertips,or a thumb and three fingertips for modifier chords. However, the fourfingertip combination is both easiest to perform and by far mostcompatible with the typing, hand resting, and pointing activities thatcan also take place on a multi-touch surface. Using the thumb and threefingertips is a bit more awkward and would necessitate cancellation whenthe pinky, or in general, a fifth digit touched the surface beforecommit by other modifiable input activity. Using three fingertips formodifier chords would necessitate cancellation when either the thumb ora fourth fingertip from the modifier chord hand touched the surfacebefore commit by modifiable activity on the opposite hand. Threefingertips modifier chords would also be incompatible with dragoperations that are preferably assigned to three-fingertips on amulti-touch surface.

Though embodiments and applications of this invention have been shownand described, it will be apparent to those skilled in the art thatnumerous further embodiments and modifications than mentioned above arepossible without departing from the inventive concepts disclosed herein.The invention, therefore, is not to be restricted except in the truespirit and scope of the appended claims.

1. A multi-touch surface apparatus that interprets 4-finger chordsperformed concurrently with other input activity as modifiers of thatinput activity, thereby avoiding awkward thumb or pinky reaches forconventional modifier keys, reducing hand strain and reducing thetendency of hands to drift off of home row, the apparatus comprising:multiple-touch sensing means that reports the locations and times offinger contacts with a surface, modifier configuration means establishesmodifier zones as horizontal bands across the surface and associateseach modifier zone with one or more modifier types; synchronizationdetection means that scans for modifier chord touchdowns consisting ofsubstantially simultaneous touches by four and no more than four digitsof a hand; modifier zone selection means that selects modifier types forthe chord according to which modifier zones said simultaneous touchesfall within; chord cancellation means that cancels a modifier chord ifthe remaining unsynchronized digit from its hand touches before othermodifiable input activity commits the chord; modifier applicabilitysearch means that commits the modifier chord and sends modifier presssignals corresponding to the chord's selected modifier types to a hostcomputer upon detection of modifiable input activity that occurssubstantially between touchdown and liftoff of the modifier chord; and,chord liftoff detection means that reports when none of the digitsoriginally constituting the modifier chord are touching the surface andsends modifier release signals to the host computer accordingly.
 2. Theapparatus of claim 1 wherein the apparatus additionally comprises atyping recognition means and surface key layout with a home row of keysupon which the hands normally rest, wherein the four digits that canconstitute a modifier chord do not include the thumb, and wherein Shiftmodifier zones are configured along the home row of keys and othermodifier types are configured for zones substantially above and belowthe home row of keys.
 3. The apparatus of claim 2 wherein any Shiftsignals arising from a committed modifier chord are temporarily releasedduring transmission to the host of keypress signals from a thumb keytouch and then re-pressed if the modifier chord has not yet lifted. 4.The apparatus of claim 1 wherein the modifier zone selection meanscompensates for the natural arch in a row of relaxed fingers bycomputing for each touch of the modifier chord a vertical offset from adefault location of the finger whose identity a contact identificationmeans has assigned to the touch.
 5. The apparatus of claim 4 wherein themodifier zone selection means simultaneously selects the modifier typesfrom each zone within which any of the simultaneous touches fall exceptif some of the simultaneous touches are bunched straddling the borderbetween two zones, in which case the modifier type associated with thesingle zone that the average of the vertical offsets lies within isselected.
 6. The apparatus of claim 1 wherein the locations of themodifier zones on the surface are indicated to the user via printing onthe surface background colors or textures unique to each zone.
 7. Theapparatus of claim 1 wherein the locations of the modifier zones on thesurface are indicated to the user via active surface display ofbackground colors or textures unique to each zone.
 8. A multi-touchsurface apparatus that interprets certain finger chords performedconcurrently with other input activity as modifiers of that inputactivity, thereby avoiding awkward thumb or pinky reaches forconventional modifier keys, reducing hand strain and reducing the needto reposition the hands, the apparatus comprising: multiple-touchsensing means that reports the locations and times of finger contactswith a surface; modifier configuration means that establishes fingerarrangement templates and associates each template with one or moremodifier types; synchronization detection means that scans for modifierchord touchdowns consisting of substantially simultaneous touches by apredetermined combination of digits of a hand; finger arrangementmatching means that selects for the chord the modifier types associatedwith the template that most closely matches the arrangement of themodifier chord touches; chord cancellation means that cancels a modifierchord if the remaining digits from its hand not included in thepredetermined combination touch before other modifiable input activitycommits the chord; modifier applicability search means that commits themodifier chord and sends modifier press signals corresponding to thechord's selected modifier types to a host computer upon detection ofmodifiable input activity that occurs substantially between touchdownand liftoff of the modifier chord; and modifier chord liftoff detectionmeans that reports when none of the digits originally constituting themodifier chord are touching the surface and sends modifier releasesignals to the host computer accordingly.
 9. The apparatus of claim 8wherein any Shift signals arising from a committed modifier chord aretemporarily released during transmission to the host of keypress signalsfrom thumb key touches and then re-pressed if the modifier chord has notyet lifted.
 10. The apparatus of claim 8 wherein four fingertipsexcluding the thumb is used as the predetermined combination of digits,the neutral four fingertip arrangement of FIG. 16A is associated withthe shift modifier type, and the horizontally spread arrangementtemplate of FIG. 16B is associated with the Ctrl modifier type.
 11. Theapparatus of claim 8 wherein the template matching means computes thematching error as the sum of squared distances between each templatepoint's offset from template center and the offset to the correspondingmodifier chord touch from chord center, thus making the matching processindependent of where the chord is performed on the surface.
 12. A methodof controlling a modifier on a multi-touch surface having a key layoutthereon, the method comprising: activating the modifier bysimultaneously dropping a plurality of fingertips of a hand into amodifier zone on or near a home row of the key layout; and maintainingactivation of the modifier by leaving at least one of the plurality offingertips in contact with the modifier zone.
 13. The method of claim 12further comprising: deactivating the modifier by removing each of theplurality of fingertips from contact with the modifier zone.
 14. Themethod of claim 12 further comprising: deactivating the modifier byplacing a thumb of the hand in contact with the key layout.
 15. Themethod of claim 12 wherein the modifier zone on or near a home row ofthe key layout may be reached through straight flexion or extension ofthe fingers from their home row position.
 16. The method of claim 12further comprising: concurrently activating a second modifier key bysimultaneously dropping a plurality of fingertips of the hand into twomodifier zones on or near a home row of the key layout, wherein thefingertips are clearly distributed into the two modifier zones.
 17. Amulti-touch surface having embodied thereon a key layout, the key layoutcomprising at least one modifier zone on or near a home row of the keylayout wherein a modifier will be activated by the simultaneoustouchdown of a plurality of fingertips within the at least one modifierzone and maintained by leaving at least one of the plurality offingertips in contact with the modifier zone.
 18. The multi-touchsurface of claim 17 comprising a plurality of modifier zones on or neara home row of the key layout, wherein each modifier zone corresponds toa different modifier.
 19. The multi-touch surface of claim 18 whereineach of the plurality of modifier zones is indicated by a differentcolor.
 20. A method of processing incoming signals from a multi-touchsurface, the method comprising: detecting modifier chords by monitoringthe incoming touch sequence for simultaneous touchdowns of a pluralityof fingertips of a hand within one or more predetermined modifier zones;and, if a new modifier chord is detected: allocating a correspondingmodifier state data structure; and appending the modifier state datastructure to a modifier chord state chain; and checking for cancellationof a modifier chord; and, if cancellation of a modifier chord isdetected: recording either a modifier chord liftoff time or modifierchord cancellation.
 21. The method of claim 20 further comprising:searching a modifier state chain for un-cancelled modifier chordsroughly coincident with modifiable input activity; and, if anyapplicable modifier chords whose signals have not already been sent to ahost computer are identified; sending the modifier signals to the hostcomputer.
 22. A multi-touch surface apparatus comprising: a multi-touchsurface that reports the locations and times of finger contacts with asurface; a synchronization detector that scans for modifier chordtouchdowns consisting of substantially simultaneous touches by aplurality of digits of a hand in a pre-defined modifier zone on themulti-touch surface and sends modifier signals accordingly; and a chordliftoff detector that reports when none of the digits originallyconstituting the modifier chord are touching the surface and sendsmodifier release signals accordingly.
 23. The multi-touch surfaceapparatus of claim 22 wherein there are a plurality of pre-definedmodifier zones on the multi-touch surface, each pre-defined modifierzone corresponding to a different modifier, the multi-touch surfaceapparatus further comprising: a modifier zone selector that selectsmodifier types for the chord according to which modifier zones thesimultaneous touches fall within.
 24. The multi-touch surface apparatusof claim 23 further comprising: a chord canceller that cancels amodifier chord if a remaining unsynchronized digit from the hand touchesthe multi-touch surface before other modifiable input activity commitsthe chord.
 25. The multi-touch surface apparatus of claim 22 furthercomprising: a chord canceller that cancels a modifier chord if aremaining unsynchronized digit from the hand touches the multi-touchsurface before other modifiable input activity commits the chord. 26.The apparatus of any of claims 22- 25 further comprising a key layout onthe multi-touch surface, the key layout having a home row wherein afirst modifier zone is configured along the home row.
 27. The apparatusof claim 26 wherein at least one additional modifier zone is configuredadjacent the home row.
 28. The apparatus of claim 27 wherein the atleast one additional modifier zone comprises a second modifier zoneabove the home row and a third modifier zone below the home row.
 29. Theapparatus of claim 28 wherein the modifiers corresponding to the first,second, and third modifier zones are selected from the group consistingof Shift, Ctrl, Alt, Windows, Open Apple, or Meta.
 30. The apparatus ofclaim 26 wherein the location of the modifier zone is indicated byprinting on the multi-touch surface.
 31. The apparatus of claim 30wherein the printing on the multi-touch surface comprises backgroundcolor.
 32. The apparatus of claim 26 wherein the location of themodifier zone is indicated by texture on the multi-touch surface. 33.The apparatus of any of claims 27- 29, wherein the locations of themodifier zones are indicated by printing on the multi-touch surface. 34.The apparatus of claim 33 wherein the printing on the multi-touchsurface comprises a unique background color for each zone.
 35. Theapparatus of claim 27- 29 wherein the locations of the modifier zonesare indicated by texture on the multi-touch surface.
 36. The apparatusof claim 26 wherein the location of the modifier zone is indicated byactive surface display of background color.
 37. The apparatus of claim26 wherein the location of the modifier zone is indicated by activesurface display of texture.
 38. The apparatus of any of claims 27- 29wherein the locations of the modifier zones are indicated by activesurface display of background colors unique to each zone.
 39. Theapparatus of claim 27- 29 wherein the locations of the modifier zonesare indicated active surface display of textures unique to each zone.40. A multi-touch surface apparatus comprising: a multi-touch surfacethat reports the locations and times of finger contacts with a surface;a synchronization detector that scans for modifier chord touchdownsconsisting of substantially simultaneous touches by a predeterminedcombination of digits of a hand in a predetermined arrangement; fingerarrangement matcher that selects a modifier corresponding to themodifier chord and sends modifier signals accordingly; and a chordliftoff detector that reports when none of the digits originallyconstituting the modifier chord are touching the surface and sendsmodifier release signals accordingly.
 41. The multi-touch surfaceapparatus of claim 40 further comprising: a chord canceller that cancelsa modifier chord if the remaining digits from its hand not include inthe predetermined combination touch before other modifiable inputactivity commits the chord.
 42. The multi-touch surface apparatus ofclaim 40 or 41 wherein the predetermined arrangement is selected fromthe group consisting of a neutral fingertip arrangement and a spreadfingertip arrangement.
 43. The multi-touch surface apparatus of claim 42wherein the modifiers associated with the predetermined arrangement areselected from the group consisting of: Shift, Ctrl, Alt, Windows, OpenApple, or Meta.